Skip to content

Refactor how we build pagination response to support collections in SQL#3521

Open
aaronburtle wants to merge 6 commits intomainfrom
dev/aaronburtle/Fix-Pagination-Column-Response
Open

Refactor how we build pagination response to support collections in SQL#3521
aaronburtle wants to merge 6 commits intomainfrom
dev/aaronburtle/Fix-Pagination-Column-Response

Conversation

@aaronburtle
Copy link
Copy Markdown
Contributor

@aaronburtle aaronburtle commented May 6, 2026

Why make this change?

Closes #3462

What is this change?

Refactors how SQL Find responses signal pagination so it no longer relies on the JSON shape of the result list.

Previously, SqlResponseHelpers.FormatFindResult appended pagination metadata to the row collection as an in-band sentinel a single-element JSON array wrapping { "nextLink": ... } (REST) or { "after": ... } (MCP) and OkResponse recovered it by checking whether the last element of the result list was a JsonValueKind.Array. This worked only because SQL backends never produced array-shaped row values.

That assumption breaks with SQL Server's JSON type, JSON_ARRAY(), vector, and other collection-shaped column values a real data row could carry an array-shaped tail value and be misclassified as a pagination envelope.

This change carries pagination metadata out-of-band:

  • FormatFindResult builds the response envelope directly as a typed projection ({ value, nextLink } for REST, { value, after } for MCP) using a bool hasNext local instead of encoding the decision into the row collection.
  • OkResponse(JsonElement, bool?) is reduced to a single-responsibility envelope wrapper. The signature is preserved (isMcpRequest is now optional and accepted-but-unused) so existing callers of Microsoft.DataApiBuilder.Core continue to compile and link unchanged. The shape-detection branch is removed from the body.
  • GetConsolidatedNextLinkForPagination is preserved with its original signature and JsonElement return type for source compatibility. Internally, FormatFindResult now constructs the next-page URL inline (it needs the URL as a string to put into the typed response envelope), avoiding a serialize-then-parse round-trip just to satisfy the old wrapper shape.
  • DetermineExtraFieldsInResponse gains a JsonValueKind.Object guard so future array/scalar/collection-typed row shapes return an empty extra-fields set rather than throwing on EnumerateObject().
  • Drive-by cleanup: removed a dead ! null-forgiving operator, standardized JsonValueKind checks on is, refreshed stale comments and XML docs, removed dead commented-out code.

Wire format is unchanged for both REST and MCP. Public API surface is byte-identical to main no signatures added, removed, or renamed; no [Obsolete] warnings emitted to downstream consumers. The new gate:

bool hasNext = isCollection && SqlPaginationUtil.HasNext(...)

is logically equivalent to the original gate.

How was this tested?

Added seven focused unit tests in SqlResponseHelpersUnitTests.cs pinning the envelope shape across every branch:

  1. Empty result
  2. Single object
  3. Collection without next page
  4. Collection with next page (REST → nextLink)
  5. Collection with next page (MCP → after)
  6. Round-trip of a row containing an array-valued column (forward-looking coverage for SQL Server JSON/vector column shapes)
  7. Regression guard for the actual pre-refactor failure mode: a result whose last top-level element is itself a JSON array the exact trigger the old in-band sentinel detection used is now correctly preserved as ordinary data instead of being unpacked as a pagination envelope.

All seven pass. dotnet format --verify-no-changes is clean across the touched files.

@aaronburtle aaronburtle self-assigned this May 6, 2026
@aaronburtle aaronburtle moved this from Todo to Review In Progress in Data API builder May 6, 2026
@aaronburtle aaronburtle added this to the May 2026 milestone May 6, 2026
@aaronburtle aaronburtle changed the title Refactor pagination reponse [DRAFT] Refactor pagination reponse May 6, 2026
@aaronburtle
Copy link
Copy Markdown
Contributor Author

@copilot can you review this PR please?

Copy link
Copy Markdown
Contributor

Copilot AI commented May 7, 2026

@copilot can you review this PR please?

I reviewed the PR changes in e0de2b2 and didn’t find blockers in the pagination refactor or the added unit coverage. I did attempt to run the targeted SqlResponseHelpersUnitTests, but this environment is missing SDK 8.0.420 from global.json, so test execution was blocked here.

@aaronburtle aaronburtle marked this pull request as ready for review May 7, 2026 09:53
Copilot AI review requested due to automatic review settings May 7, 2026 09:53
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR refactors REST/MCP pagination response shaping so pagination metadata (nextLink for REST, after for MCP) is returned out-of-band in the response envelope instead of being encoded into the row array, addressing issue #3462 around ambiguity with array-shaped JSON payloads.

Changes:

  • Refactors SqlResponseHelpers.FormatFindResult to build paginated envelopes directly ({ value, nextLink } / { value, after }) and simplifies OkResponse to only wrap results in { value: [...] }.
  • Renames/changes the pagination next-link builder in SqlPaginationUtil to return a plain string URI.
  • Adds focused unit tests to pin envelope shapes for empty/single/collection/paginated responses and an attempted regression guard.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File Description
src/Service.Tests/UnitTests/SqlResponseHelpersUnitTests.cs Adds unit tests for FormatFindResult envelope shapes and pagination metadata behavior.
src/Core/Resolvers/SqlResponseHelpers.cs Moves pagination metadata out-of-band; simplifies OkResponse; adds guard in extra-field detection.
src/Core/Resolvers/SqlPaginationUtil.cs Replaces JSON-element nextLink builder with a string-returning URI builder and updates docs.

Comment thread src/Core/Resolvers/SqlPaginationUtil.cs
Comment thread src/Core/Resolvers/SqlResponseHelpers.cs
Comment thread src/Service.Tests/UnitTests/SqlResponseHelpersUnitTests.cs
@aaronburtle aaronburtle changed the title [DRAFT] Refactor pagination reponse Refactor how we build pagination response to support collections in SQL May 7, 2026
@aaronburtle
Copy link
Copy Markdown
Contributor Author

@copilot I addressed your comments, can you do another review please?

Copy link
Copy Markdown
Contributor

Copilot AI commented May 7, 2026

@copilot I addressed your comments, can you do another review please?

Reviewed again on the latest commits (including e347cab). The previous API-compat concerns are addressed (signatures restored), and the regression coverage now includes the top-level array-tail sentinel case. I don’t see new blockers. I re-ran the targeted SqlResponseHelpersUnitTests here, but this environment still cannot execute dotnet test because global.json requires SDK 8.0.420 and only 8.0.419 is installed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

Status: Review In Progress

Development

Successfully merging this pull request may close these issues.

Possible bug related to new Array data types with Pagination

5 participants